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The role of evaluation in the development of 


evolutionary procedures is briefly déScribed and highlighted. Four 
aspects of evaluation technigue which distinguish efficient from 


inefficient CAI programs are identified. 


Evaluation of products is 


also characterized. Findings of a ccntinuing survey of students via 


questionnaire as to the value of certain courseware are listed. 
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Evaluation is a pervasive part of any slanntes or decision.making pro-} 
cedure, It is particularly important in procedures which are evolutionary 
in nature singe such procedures tend to require decisions which stank he 

_made on the basis of predetermined rules, The fact that these decisions . 
generally must ba made rapidly and that they frequontly reouire dotailed — 
knowledge ofa host of tang: al issues means that most formative evaluation 
is carried out by caiba directly involved in the work rather than by a central 
administrator or a professional evaluater, Such decentralized operations can 
be quite difficult to describe, much-less evaluate, from outsides with any 
Gegree cf accuracy. Unfortunately, in current CAI dovelopment, it,is the 
functioning of these decentralized processes that is often of more interest 
than their end products, For example, tho procedures followed in the effi- 
cient production of effective instructional meterials can rarcly be inferred 
from a summative evaluation, of tha completed lesson material. Yet, these 
processes are likely to be of far more general interest than the products. 
Periodic interviews or logs can provide some information on process cata, 

= Tigh frequently it is feheaasia for an evaluator to become immersed in the. 
process itself before an accurate ane can be obtained (Rippey, 1974; 
House, 1974). For instance, sinitertes of ad independent PLATO curricular 
deveecnin efforts over the past years has tndioaked four basic areas dn 
which effective groups appear to differ from inefficient or irieffactive 

groups.’ . wv 

1) Members of effective’ groups have or rapidly develop experience 


An individualized instruction and in the relevant subject- 
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2) Effective groups tend to develop or adopt standardized procedures | 
and techniques only after direct instructional experience 
indicates appropriateness to the aims of the group. Effective Ya 
groups also make significantly greater use of std plawdtend 
. techniques. ‘ . 
3) Effective groups identify potential problems earlier and make 
more active efforts to gather information and experience needed 
to overcome such problems, For example, effective groups tend : s 
to use student and peer evaluation of preliminary instructional | 
- designs earlier and more extensively than ineffective a 
4) Effective groups tend to use all aiaiceca'ian evaluative iaroenna v 
tion as soon as possible for ‘modification of inoteusbiond 
materials. Thus, while both effective and ineffective groups 
may gather on-line student response data, ineffective groups 
tend to simply store the information without processing it or 
use it solely for production of administrative paporta, 
In ail but the first of these four areas, information from interviews, 
logs, or progress reports is usually insufficient to dievenpisen effective 
groups from ineffective groups. Only direct, frequent, and unobtrusive 
observation can determine whether a described operation was performed compe- 
tently or in a perfunctory and unproductive manner. Unfor sunately such 
extensive observation is very costly in resources and is effective only where ~ 
the outside evaluator is perceived as, and “ fact is, interested in describing 
the process and in improving the group's effectiveness rather than in passing 
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judgment. 


. te 
White process evaluation is currently of major interest within the PLATO 
eeHe, product evalvation cannot be overlooked. Here also, an evalationay 
development program produces complications as well as opportunities. A basic 


. 


' conflict results from the simtaneous operation of PLATO both as a service 
and asa ediasen facility. The evolutionary approach demands, for example, 
that exporience be gathered in a setting as close as possible to the current 
concept of an ultimate application of the system. Thus, PLATO must be™vsed 
extensively as if it were a final product devoted solely to instruction in 
order that realistic data be gathered. At the same time, changes are 
continually being made to PLATO as a result of those data. Among the 
invepininencs that arise are questions sich as: "To what extent are summa. 
tive evaluations of products sieh as courseware affected by presentation via 
a donétantly whesging medion?! Answers to such questions must be gathered 

“through aian and contimous contact with the current consumers: the students 
and ‘instructors. Among the findings to date: , 

1) Students are able to distinguish reliably between limitations 
En of the medium and courseware. 
- 2) Initial student attitudes toward both courseware and medium are 
heavily influenced by the attitudos of their instructors, 
33 Meetatades toward the nedium stabilize within the first few hours 
of exposure and are generally favorable. . 

4) HORE HOEK TANS student comments have to do with factors which 

impede their progress through the instructional sequences, 

e This includes such problems as system unreliability, poorly 

written course materials, and lengthy attitude questionnaires, - 


Unfortunately, the most. rapid and reliable guage of student and instruc. 


tor attitudes is provided by their response to frequent questionnaires and 
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interviews. To maintain this flow of information without either alienating 
the source or taking up excessive quantities of time, item sampling tech- 
niques used in commercial consumer attitude survoys have been adopted, 

The full range ‘of applications of evaluation procedures in an evolution- 
ary development and their results can only be sivpended by the brief descrip- 
tas above, It should, however,. be clear that such a SEUEENE provides 
gratifying evidence for evaluators who have held that. the evaluation process 
should be considered as an integral part of educational research and develop. 
ment rather than a process which is preformed after everythirg else is 
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